Evaluation of a Video Generation Method Linking Dance and Scenes

Owing to the impact of COVID-19, the venues for dancers to perform have shifted from the stage to the media. In this study, we focus on the creation of dance videos that allow audiences to feel a sense of excitement without disturbing their awareness of the dance subject and propose a video generation method that links the dance and the scene by utilizing a sound detection method and an object detection algorithm. The generated video was evaluated using the Semantic Differential method, and it was confirmed that the proposed method could transform the original video into an uplifting video without any sense of discomfort. © 2023 SPIE.

Video-Based Online Interactive System Using CNN Segmentation Method

In recent years, the COVID-19 has made it difficult for people to interact with each other face-to-face, but various kinds of social interactions are still needed. Therefore, we have developed an online interactive system based on the image processing method, that allows people in different places to merge the human region of two images onto the same image in real-time. The system can be used in a variety of situations to extend its interactive applications. The system is mainly based on the task of Human Segmentation in the CNN (convolution Neural Network) method. Then the images from different locations are transmitted to the computing server through the Internet. In our design, the system ensures that the CNN method can run in real-time, allowing both side users can see the integrated image to reach 30 FPS when the network is running smoothly. © 2023 IEEE.

DenseNet201 Model for Robust Detection on Incorrect Use of Mask

In 2022, the COVID-19 pandemic is still occurring. One of the optimal prevention efforts is to wear a mask properly. Several previous studies have classified the use of masks incorrectly. However, the accuracy resulting from the classification process is not optimal. This research aims to use the transfer learning method to achieve optimal accuracy. In this research, we used three classes, namely without a mask, incorrect mask, and with a mask. The use of these three classes is expected to be more detailed in detecting violations of the use of masks on the face. The classification method used in this research uses transfer learning as feature extraction and Global Average Pooling and Dense layers as classification layers. The transfer learning models used in this research are MobileNetV2, InceptionV3, and DenseNet201. We evaluate the three models' accuracy and processing time when using video data. The experimental results show that the DenseNet201 model achieves an accuracy of 93%, but the processing time per video frame is 0.291 s. In contrast to the MobileNetV2 model, which produces an accuracy of 89% and the processing speed of each video frame is 0.106 s. This result is inversely proportional to accuracy and speed. The DenseNet201 model produces high accuracy but slow processing time, while the MobileNetV2 model is less accurate but has faster processing. This research can be applied in the crowd center to monitor health protocols in the use of masks in the hope of inhibiting the transmission of the COVID-19 virus. © 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

From Art Image to Video/Comic-Image Learning. The Video Spot Stay at Home

From the Covid-19 health emergency entered our lives, the web continues to alleviate moments of isolation with ironic memes, photos and videos that, despite having been considered an irreverence to the masterpieces of Art and/or one of the many uses of irony to exorcise fear, they have favored the staging of video-graphic products with a strong ‘humor' component. Within these premises, in the context of graphic design, this paper will evaluate aspects as the analysis of fashion environment as expressive language of living indoor during Covid-19 pandemic;the audiovisual languages and compositional criteria for the creation and multimedia communication of a video-graphic spot on Stay at home communication campaign. The video-graphic products were analyzed on the basis of: relationship between ‘humor' message and supporting artwork;integration between image and photo-cinematography;figurative languages generative of graphic signs;duration of audiovisual spot;sound component as key to emotional reading;communication strategies. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Object detection and activity recognition in video surveillance using neural networks

Purpose: This paper aims to implement and extend the You Only Live Once (YOLO) algorithm for detection of objects and activities. The advantage of YOLO is that it only runs a neural network once to detect the objects in an image, which is why it is powerful and fast. Cameras are found at many different crossroads and locations, but video processing of the feed through an object detection algorithm allows determining and tracking what is captured. Video Surveillance has many applications such as Car Tracking and tracking of people related to crime prevention. This paper provides exhaustive comparison between the existing methods and proposed method. Proposed method is found to have highest object detection accuracy. Design/methodology/approach: The goal of this research is to develop a deep learning framework to automate the task of analyzing video footage through object detection in images. This framework processes video feed or image frames from CCTV, webcam or a DroidCam, which allows the camera in a mobile phone to be used as a webcam for a laptop. The object detection algorithm, with its model trained on a large data set of images, is able to load in each image given as an input, process the image and determine the categories of the matching objects that it finds. As a proof of concept, this research demonstrates the algorithm on images of several different objects. This research implements and extends the YOLO algorithm for detection of objects and activities. The advantage of YOLO is that it only runs a neural network once to detect the objects in an image, which is why it is powerful and fast. Cameras are found at many different crossroads and locations, but video processing of the feed through an object detection algorithm allows determining and tracking what is captured. For video surveillance of traffic cameras, this has many applications, such as car tracking and person tracking for crime prevention. In this research, the implemented algorithm with the proposed methodology is compared against several different prior existing methods in literature. The proposed method was found to have the highest object detection accuracy for object detection and activity recognition, better than other existing methods. Findings: The results indicate that the proposed deep learning–based model can be implemented in real-time for object detection and activity recognition. The added features of car crash detection, fall detection and social distancing detection can be used to implement a real-time video surveillance system that can help save lives and protect people. Such a real-time video surveillance system could be installed at street and traffic cameras and in CCTV systems. When this system would detect a car crash or a fatal human or pedestrian fall with injury, it can be programmed to send automatic messages to the nearest local police, emergency and fire stations. When this system would detect a social distancing violation, it can be programmed to inform the local authorities or sound an alarm with a warning message to alert the public to maintain their distance and avoid spreading their aerosol particles that may cause the spread of viruses, including the COVID-19 virus. Originality/value: This paper proposes an improved and augmented version of the YOLOv3 model that has been extended to perform activity recognition, such as car crash detection, human fall detection and social distancing detection. The proposed model is based on a deep learning convolutional neural network model used to detect objects in images. The model is trained using the widely used and publicly available Common Objects in Context data set. The proposed model, being an extension of YOLO, can be implemented for real-time object and activity recognition. The proposed model had higher accuracies for both large-scale and all-scale object detection. This proposed model also exceeded all the other previous methods that were compared in extending and augmenting the object detection to activity recognition. The proposed model resulted in the highest accuracy for car crash detection, fall detection and social distancing detection. © 2023, Emerald Publishing Limited.

Fighting for a future free from violence: A framework for real-time detection of "Signal for Help”

In April 2020, by the start of isolation all around the world to counter the spread of COVID-19, an increase in violence against women and kids has been observed such that it has been named The Shadow Pandemic. To fight against this phenomenon, a Canadian foundation proposed the "Signal for Help” gesture to help people in danger to alert others of being in danger, discreetly. Soon, this gesture became famous among people all around the world, and even after COVID-19 isolation, it has been used in public places to alert them of being in danger and abused. However, the problem is that the signal works if people recognize it and know what it means. To address this challenge, we present a workflow for real-time detection of "Signal for Help” based on two lightweight CNN architectures, dedicated to hand palm detection and hand gesture classification, respectively. Moreover, due to the lack of a "Signal for Help” dataset, we create the first video dataset representing the "Signal for Help” hand gesture for detection and classification applications which includes 200 videos. While the hand-detection task is based on a pre-trained network, the classifying network is trained using the publicly available Jesture dataset, including 27 classes, and fine-tuned with the "Signal for Help” dataset through transfer learning. The proposed platform shows an accuracy of 91.25% with a video processing capability of 16 fps executed on a machine with an Intel i9-9900K@3.6 GHz CPU, 31.2 GB memory, and NVIDIA GeForce RTX 2080 Ti GPU, while it reaches 6 fps when running on Jetson Nano NVIDIA developer kit as an embedded platform. The high performance and small model size of the proposed approach ensure great suitability for resource-limited devices and embedded applications which has been confirmed by implementing the developed framework on the Jetson Nano Developer Kit. A comparison between the developed framework and the state-of-the-art hand detection and classification models shows a negligible reduction in the validation accuracy, around 3%, while the proposed model required 4 times fewer resources for implementation, and inference has a speedup of about 50% on Jetson Nano platform, which make it highly suitable for embedded systems. The developed platform as well as the created dataset are publicly available. © 2022

Ensemble Deep Learning Approach with Attention Mechanism for COVID-19 Detection and Prediction

New coronavirus (COVID-19), which first appeared in Wuhan City and is now rapidly disseminating worldwide, may be predicted, diagnosed, and treated with the help of cutting-edge medical technology, such as artificial intelligence and machine learning algorithms. To detect COVID-19, we suggested an Ensemble deep learning method with an attention mechanism. The suggested approach uses an ensemble of RNN and CNN to extract features from data from diverse sources, such as CT scan pictures and blood test results. For image and video processing, CNNs are the most effective. RNNs, on the other hand, use text and speech data to extract features. Further, an attention mechanism is used to determine which features are most relevant for classification. Finally, the deep learning network utilizes the selected features for detection and prediction. As a result, data can be used to forecast future medical needs. © 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

SlowFast GCN Network for Quantification of Parkinsonian Gait Using 2D Videos

Parkinson's disease(PD) is a progressive neu-rodegenerative disease defined by clinical syndrome including bradykinesia, tremor and postural instability. The PD-related disability and impairment are usually monitored by clinicals using the MDS-UPDRS scale. However, due to COVID-19, it became much harder for the patients to reach hospitals and obtain necessary assessment and treatment. Nowadays, 2D videos are easily accessible and can be a promising so-lution for on-site and remote diagnosis of movement disorder. Inspired by the frequency-based video processing mechanism of human visual system, we propose a video-based SlowFast GCN network to quantify the gait disorder. The model consists of two parts: the fast pathway and the slow pathway. The former detects characteristics such as tremor and bilateral asymmetry, while the latter extracts characteristics such as bradykinesia and freezing of gait. Furthermore, in order to investigate the influence of age on the model performance, an aged control group and a young control group were set up for verification. The proposed model was evaluated on a video dataset collected from 68 participants. We achieved a balanced accuracy of 87.5% and precision of 87.9%, which outperformed existing competing methods. When replacing the young healthy controls with the same number of older controls, the balanced accuracy and precision were decreased by 10.4% and 9.7%, which indicates that age has a significant effect on the model perfomance. © 2022 IEEE.

VVC intra prediction decoder: Feature improvement and performance analysis

Nowadays, streaming applications have been in great demand, especially due to covid-19 (teleworking, online teaching, virtual reality, etc.). In addition, artificial intelligence has become widely used especially in video processing domains, so a video with high quality improves the accuracy rate of this application. To meet these needs, the Versatile Video Coding standard (VVC) has appeared to give a high compression efficiency compared to high-efficiency video coding. This norm consists of a high complexity algorithm that offers an improvement in processing time and decreases the bit rate by 50 % thanks to several new compression techniques. In this context, we propose the implementation of an intra prediction decoding chain of this standard on a system on chip. In this work, we highlight the VVC feature enhancements, we present the suitable method for VVC intra-prediction decoder implementation on the PYNQ-Z2, and we provide profiling in terms of decoding time and power consumption. As a future work, this study is helpful to distinguish the block that will be a candidate for a Hardware acceleration. © 2022 IEEE.

Real-Time Masked Face Recognition Using FaceNet and Supervised Machine Learning

The coronavirus pandemic has led to the implementation of health protocols such as the use of masks worldwide. Without exception, work activities also require the wearing of masks. This condition makes it difficult to recognize an individual's identity because the mask covers half of the face, especially when the employee is present. The attendance system recognizes a face without a mask more accurately, in contrast, a masked face makes identity recognition inaccurate. Therefore, this study proposes a combination of facial feature extraction using FaceNet and several classification methods. The three supervised machine learning methods were evaluated, namely multiclass Support Vector Machine (SVM), K-Nearest Neighbor, and Random Forest. Furthermore, the masked face recognition system was evaluated using real-time video data to assess the accuracy and processing time of the video frame. The accuracy result on real-time video data using a combination of FaceNet with K-NN, multiclass SVM, or Random Forest of 96.03%, 96.15%, and 54.04% are obtained respectively and in processing time per frame of 0.056 s, 0.055 s, and 0.061 are obtained respectively. The results show that the combination of the FaceNet feature extraction method with multiclass SVM produces the best accuracy and data processing speed. In other words, this combination can reach 18 fps at real-time video processing. Based on these results, the proposed combined method is suitable for real-time masked face recognition. This study provides an overview of the masked face recognition method so that it can be a reference for the contactless attendance system in this pandemic era. © 2022, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

Conventional versus learning-based video coding benchmarking: Where are we?

Video coding technology and standards have largely shaped the current world in many domains, notably personal communications and entertainment, as demonstrated during recent COVID-19 times. AI-based multimedia tools already have a major impact in many computer vision tasks, even reaching above human performance, and are now arriving to multimedia coding. In this context, this paper offers one of the first extensive benchmarking of deep learning-based video coding solutions regarding the most powerful and optimized conventional video coding standard, the Versatile Video Coding standard, under the solid, meaningful, and extensive JVET common test conditions. This study will allow the video coding research community to know the current status quo in this emerging ‘battle’ between learning-based and conventional video coding solutions and better design future developments. © 2022 SPIE.

AI-Based Saliency-Aware Video Coding

The demand for video through over-the-top (OTT) has been constantly increasing in recent years. During the COVID-19 pandemic, demand skyrocketed, hence leading to the need for better video compression. The human visual system (HVS) can quickly select visually important regions in its visual field. These regions are captured at high resolution, while other peripheral regions receive little attention. Saliency maps are a way to imitate the HVS attention mechanism. Recently, deep learning-based saliency models have achieved tremendous improvements. This article leverages state-of-the-art deep learning-based saliency models to improve video coding efficiency. First, a saliency-based rate control scheme is integrated in a high-efficiency video encoder (HEVC). Then, a saliency-guided preprocessing filtering step is introduced. Finally, the two approaches are combined. Objective and subjective evaluations show that it can lower the bitrate from 6% to almost 30% while maintaining the same visual quality. © 2002 Society of Motion Picture and Television Engineers, Inc.

Measuring Accuracy Towards Facial Video Heart-Rate Estimation Using Haar-Cascade and CNN Method

COVID-19 leads us to have a social distancing even for health-treatment. In this study, we attempt to estimate heart rates in humans using camera-based remote photoplethysmography (rPPG) methods, which are named after conventional PPG methods. The basic concept is focused on capturing minute variations in skin color during the human body's cardiac cycle, which involves the inflow and outflow of blood from the heart to other body parts. We have compared the performance of different methods of Blind Source Separation and face detection which form an integral part in accurately calculating the heart rate. Purpose: The purpose of this method was comparing the actual heart rate with a tuned parameter of Face Video Heart Rate estimation with CNN and OpenCV haar-cascade. Patients and methods: Videos in the dataset are run through a face detection model to get the region of interest for heart rate calculation. Source signals are converted to frequency domain for filtering and peak detection to obtain heart rate estimates Results: Face segmentation using Convolution Neural Network gives better results than the Haar Cascade OpenCV face detection module, which is as expected. Conclusion: Face segmentation using Convolution Neural Network gives better results than the Haar Cascade OpenCV face detection module. CNNs are slower to detect faces than the Open-CV module. Choosing an ROI by segmenting out facial pixels helped to keep the outliers low and therefore increased the robustness. © 2022 IEEE.

State of art on object detection solution applied to COVID 19's spreading prevention

In order to effectively prevent the spread of COVID19, people from different parts of the world were supposed to be wearing face masks after the WHO put it as a primordial instruction to stop its propagation. Researchers from different backgrounds gathered their efforts to ensure the respect of wearing face mask, namely AI field researchers. In this research, we are interested on the AI applications that were done from the beginning of the pandemic to prevent the COVID 19 contamination, especially those related to the mask wearing detection. The detection of wearing mask is classified as a computer vision problem, more specifically, an object detection one. Besides, with the evolution of the computational power and the availability of huge number of datasets, deep learning models using image and video processing techniques were proposed in order to detect people transgressing the wearing mask rule. In this paper we introduce a literature review of object detection, a case study of this problem which consists in the wearing mask detection, the related works as well as the different proposed solutions, and the suggested general pipeline for the treatment of this problem. © 2021 IEEE.

PER-COVID (PEople pRoximity based on Certified and coOperative VIDeo-intelligence): A Software Tool for Physical Distancing and PPE Monitoring

The recent COVID-19 pandemic has led to a growing interest in IT tools for monitoring social distance and for checking the presence of personal protective equipment and whether it is worn properly. Correct monitoring in outdoor and indoor areas is essential to limit the spread of the virus and the risk of being infected. This paper presents PER-COVID, a software platform capable of monitoring crowds of people and the correct use of personal protective equipment in real time using innovative computer vision algorithms. The proposed system architecture and functional characteristics are illustrated, as well as some user interface screens are provided for simple interpretation and monitoring of critical events. © 2021 IEEE.

Design and Development of an Integrated Internet of Audio and Video Sensors for COVID-19 Coughing and Sneezing Recognition

There are a lot of ongoing efforts to combat the COVID-19 pandemic using different combinations of low-cost sensing technologies, information/communication technologies, and smart computation. To provide COVID-19 situational awareness and early warnings, a scalable, real-time sensing solution is needed to recognize risky behaviors in COVID-19 virus spreading such as coughing and sneezing. Various coughing and sneezing recognition methods use audio-only or video-only sensors and Deep Learning (DL) algorithms for smart event recognition. However, each of these recognition processes experiences several types of failure behaviors due to false detection. Sensor integration is a solution to overcome such failures. Moreover, it improves event recognition precision. With the wide availability of low-cost audio and video sensors, we proposed a real-time integrated Internet of Things (IoT) architecture to improve the results of coughing and sneezing recognition. Implemented architecture joins edge and cloud computing. In edge computing, the microphone and camera are connected to the internet and embedded with a DL engine. Audio and video streams are fed to edge computing to detect coughing and sneezing actions in realtime. Cloud computing, which is developed based on the Amazon Web Service (AWS), combines the results of audio and video processing. In this paper, a scenario of a person coughing and sneezing was developed to demonstrate the capabilities of the proposed architecture. The experimental results show that the proposed architecture improved the reliability of coughing and sneezing recognition in the integrated cloud system compared to audio-only and video-only detectors. Three factors have been considered to compare the results of the proposed architecture: F-score, precision, and recall. The precision and recall of the cloud detector are improved on average by %43 and %15, respectively, compared to audio-only and video-only detectors. The F-score improved on average 1.24 times. © 2021 IEEE.

An Imaging system for teleteaching support using multi-view video shooted by basketball players

After the COVID-19 pandemic, the importance of teleteaching using networks has been increasing in regular university classes. The authors have been studying on the improvement of athletic performance of college women's basketball as extracurricular activities using information processing from multi-image. In this paper, we propose the use of such a multi-visual system in basketball classes. In this experiment basketball athletes at the gymnasium gallery shooted play of other athletes on the court using smartphones. Furthermore, the effectiveness of this video is discussed in terms of regular university classes. One of the characteristics of this video information system, which was realized on the premise of the existing campus network, is that it is a highly feasible system that does not interfere with other lectures and the daily work of faculty members in the university. © 2021 IEEE.

Sparse Crowd Flow Analysis of Tawaaf of Kaaba During the COVID-19 Pandemic

The advent of the COVID-19 pandemic has adversely affected the entire world and has put forth high demand for techniques that remotely manage crowd-related tasks. Video surveillance and crowd management using video analysis techniques have significantly impacted today's research, and numerous applications have been developed in this domain. This research proposed an anomaly detection technique applied to Umrah videos in Kaaba during the COVID-19 pandemic through sparse crowd analysis. Managing the Kaaba rituals is crucial since the crowd gathers from around the world and requires proper analysis during these days of the pandemic. The Umrah videos are analyzed, and a system is devised that can track and monitor the crowd flow in Kaaba. The crowd in these videos is sparse due to the pandemic, and we have developed a technique to track the maximum crowd flow and detect any object (person) moving in the direction unlikely of the major flow. We have detected abnormal movement by creating the histograms for the vertical and horizontal flows and applying thresholds to identify the non-majority flow. Our algorithm aims to analyze the crowd through video surveillance and timely detect any abnormal activity to maintain a smooth crowd flow in Kaaba during the pandemic. © 2022 Tech Science Press. All rights reserved.

Drone vs. Bird Detection: Deep Learning Algorithms and Results from a Grand Challenge.

Adopting effective techniques to automatically detect and identify small drones is a very compelling need for a number of different stakeholders in both the public and private sectors. This work presents three different original approaches that competed in a grand challenge on the "Drone vs. Bird" detection problem. The goal is to detect one or more drones appearing at some time point in video sequences where birds and other distractor objects may be also present, together with motion in background or foreground. Algorithms should raise an alarm and provide a position estimate only when a drone is present, while not issuing alarms on birds, nor being confused by the rest of the scene. In particular, three original approaches based on different deep learning strategies are proposed and compared on a real-world dataset provided by a consortium of universities and research centers, under the 2020 edition of the Drone vs. Bird Detection Challenge. Results show that there is a range in difficulty among different test sequences, depending on the size and the shape visibility of the drone in the sequence, while sequences recorded by a moving camera and very distant drones are the most challenging ones. The performance comparison reveals that the different approaches perform somewhat complementary, in terms of correct detection rate, false alarm rate, and average precision.

A multi-class COVID-19 segmentation network with pyramid attention and edge loss in CT images.

At the end of 2019, a novel coronavirus COVID-19 broke out. Due to its high contagiousness, more than 74 million people have been infected worldwide. Automatic segmentation of the COVID-19 lesion area in CT images is an effective auxiliary medical technology which can quantitatively diagnose and judge the severity of the disease. In this paper, a multi-class COVID-19 CT image segmentation network is proposed, which includes a pyramid attention module to extract multi-scale contextual attention information, and a residual convolution module to improve the discriminative ability of the network. A wavelet edge loss function is also proposed to extract edge features of the lesion area to improve the segmentation accuracy. For the experiment, a dataset of 4369 CT slices is constructed, including three symptoms: ground glass opacities, interstitial infiltrates, and lung consolidation. The dice similarity coefficients of three symptoms of the model achieve 0.7704, 0.7900, 0.8241 respectively. The performance of the proposed network on public dataset COVID-SemiSeg is also evaluated. The results demonstrate that this model outperforms other state-of-the-art methods and can be a powerful tool to assist in the diagnosis of positive infection cases, and promote the development of intelligent technology in the medical field.